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Abstract. In this lecture we present a discussion of generalized statistics based on 
Renyi's, Fisher's and Tsallis's measures of information. The unifying conceptual frame- 
work which we employ here is provided by information theory. Important applications 
of generalized statistics to systems with (multi-)fractal structure are examined. 

1 Introduction 

One of the important approaches to statistical physics is provided by informa- 
tion theory erected by Claude Shannon in the late 1940s. Central tenet of this 
approach lies in a construction of a measure of the "amount of uncertainty" 
inherent in a probability distribution PQ . This measure of information (or Shan- 
non's entropy) quantitatively equals to the number of binary (yes/no) questions 
which brings us from our present state of knowledge about the system in ques- 
tion to the one of certainty. The higher is the measure of information (more 
questions to be asked) the higher is the ignorance about the system and thus 
more information will be uncovered after an actual measurement. Usage of Shan- 
non's entropy is particularly pertinent to the Bayesian statistical inference where 
one deals with the probability distribution assignment subject to prior data one 
possess about a system |2I3| . Here the prescription of maximal Shannon's en- 
tropy (i.e., maximal ignorance) - MaxEnt, subject to given constraints yields 
the least biased probability distribution which naturally protects against con- 
clusions which are not warranted by the prior data. In classical MaxEnt the 
maximum-entropy distributions are always of an exponential form and hence 
the name generalized canonical distributions. Note that MaxEnt prescription, in 
a sense, resembles the principle of minimal action of classical physics as in both 
cases extremization of a certain functionals - the entropy or action functionals - 
yields physical predictions. In fact, the connection between information and the 
action functional was conjectured by E.T. Jaynes [3] and J. A. Wheeler and 
most recently this line of reasonings has been formalized e.g., by B.R. Friedcn 
in his "principle of extreme physical information - EPF [5]. 

On a formal level the passage from information theory to statistical ther- 
modynamics is remarkably simple. In this case a maximal-entropy probability 
distribution subject to constraints on average energy, or constant average energy 
and number of particles yields the usual canonical or grand-canonical distribu- 
tions of Gibbs, respectively. Applicability in physics is, however, much wider. 
Aside of statistical thermodynamics, MaxEnt has now become a powerful tool 
in non-equilibrium statistical physics |S] and is equally useful in such areas as 
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astronomy, geophysics, biology, medical diagnosis and economics. For the latest 
developments in classical MaxEnt the interested reader may consult ref. |7| and 
citations therein. 

As successful as Shannon's information theory has been, it is clear by now 
that it is capable of dealing with only a limited class of systems one might hope 
to address in statistical physics. In fact, only recently it has become appar- 
ent that there are many situations of practical interest requiring more "exotic" 
statistics which does not conform with the generalized canonical prescription 
of the classical MaxEnt (often referred as Boltzmann-Gibbs statistics). Perco- 
lation, polymers, protein folding, critical phenomena or stock market returns 
provide examples. On the other hand, it cannot be denied that MaxEnt ap- 
proach deals with statistical systems in a way that is methodically appealing, 
physically plausible and intrinsically nonspeculative (i.e., MaxEnt invokes no 
hypotheses beyond the sample space and the evidence that is in the available 
data). It might be therefore desirable to inspect the axiomatics of Shannon's 
information theory to find out whether some "plausible" generalization is pos- 
sible. If so, such an extension could provide a new conceptual frame in which 
generalized measures of information (i.e., entropies) could find their theoretical 
justification. The additivity of independent mean information is the most natu- 
ral axiom to attack. On this level three modes of reasoning can be formulated. 
One may either keep the additivity of independent information but utilize more 
general definition of means, or keep the the usual definition of linear means but 
generalize the additivity law or combine both these approaches together. 

In the first case the crucial observation is that the most general means com- 
patible with Kolmogorov's axioms of probability are the so called quasi-linear 
means which are implemented via Kolmogorov-Nagumo functions 8 . This ap- 
proach was pioneered by A. Renyi [5], J. Aczel and Z. Daroczy in 60s and 
70s. The corresponding measure of information is then called Renyi's entropy. 
Because the independent information are in this generalization still additive and 
because the quasi-linear means basically probe dimensionality of the sample 
space one may guess that this theory should play an important role in classical 
information-theoretical systems with a non-standard geometry, such as fractals, 
multi-fractals or systems with embedded self-similarity. These include phase 
transitions and critical phenomena, chaotic dynamical systems with strange at- 
tractors, fully developed turbulence, hadronic physics, cosmic strings, etc. 

Second case amounts to a modification of the additivity law. Out of the infin- 
ity of possible generalizations the so called g-additivity prescription has found 
widespread utility. The q-calculus was introduced by F. Jackson JT] in 20 's 
and more recently developed in the framework of Quantum Groups by V. Drin- 
feld T2J and M. Jimbo ^3]. With the help of g-calculus one may formalize the 
entire approach in an unified manner by defining g-derivative (Jackson deriva- 
tive), (/-integration (Jackson integral), (/-logarithms, q-exponentials, etc. The 
corresponding measure of information is called Tsallis or non-extensive entropy. 
The g-additivity is in a sense minimal generalization because the non-additive 
part is proportional to both respective information and is linearly parametrized 
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by the only one "coupling" constant. The non-additivity prescription might be 
understood as a claim that despite given phenomena being statistically inde- 
pendent there still might be non-vanishing correlation between them and hence 
information might get "entangled". One may thus expect that Tsallis entropy 
should be important in systems with long-range correlations or long-time mem- 
ories. One may even guess that quantum non-locality might become crucial 
playground for non-extensive statistics. 

Third generalization is still not explored in the literature. It can be expected 
that it should become relevant in e.g., critical phenomena in a strong quantum 
regime. The latter can be found, for instance, in the early universe cosmolog- 
ical phase transitions or in currently much studied quantum phase transitions 
(frustrated spin systems or quantum liquids being examples). 

The structure of this lecture is the following: In Sections|21we review the basic 
information-theoretic setup for Renyi's entropy. We show its relation to (multi- 
fractal systems and illustrate how the Renyi parameter is related to multifractal 
singularity spectrum. The connection of Renyi's entropy with Fisher information 
and the metric structure of statistical manifolds (i.e., Fisher-Rao metric) are also 
discussed. In Section[3]the information theoretic rationale of Tsallis entropy are 
presented. 

2 Renyi's entropy 

Renyi entropies (RE) were introduced into mathematics by A. Renyi |U] in mid 
60's. The original motivation was strictly formal. Renyi wanted to find the most 
general class of information measures which preserved the additivity of statisti- 
cally independent systems and were compatible with Kolmogorov's probability 
axioms. 

Let us assume that one observes the outcome of two independent events with 
respective probabilities p and q. Additivity of information then requires that the 
corresponding information obeys Cauchy's functional equation 



Therefore, aside from a multiplicative factor, the amount of information received 
by learning that an event with probability p took place must be 



Here the normalization was chosen so that ignorant's probability (i.e. p — 1/2) 
sets the unit of information - bit. Formula is known as Hartley's informa- 
tion measure [2] ■ In general, if the outcomes of some experiment are A± , . . . , A n 
with respective probabilities p\ , . . . , p n , and if Ak outcome delivers Ik bits of 
information then the mean received information reads 



l(pq)=I(p)+I(q). 



(1) 



log 2 P ■ 



(2) 





4 P. Jizba 



Here g is an arbitrary invertible function - Kolmogorov-Nagumo function. The 
mean denned in Eq.Q is the so called quasi-linear mean and it constitutes the 
most general mean compatible with Kolmogorov's axiomatics |8ll4j . Renyi then 
proved that when the postulate of additivity for independent events is applied to 
Eq.© it dramatically restricts the class of possible g's. In fact, only two classes 
are possible; g(x) = cx + d which implies the Shannon information measure 

n 

ZW = -$>*log 2 (Pk), ( 4 ) 

k=l 

and g(x) — c 2^ q ^ x + d which implies 

T ^ V) = JT~qj (Pti) ' (5) 

with q > (c and d are arbitrary constants). In both cases V = {p\, . . . ,p n }. 
Note that for linear g's the quasi-linear mean turns out to be the ordinary linear 
mean and hence Shannon's information is the averaged information in the usual 
sense. Information measure defined by (JHJ) is called Renyi's information measure 
(of order q) or Renyi's entropy. Term "entropy" is chosen in a close analogy 
with Shannon's theory because Renyi's entropy also represents the disclosed 
information (or removed ignorance) after performed experiment. On a deeper 
level it might be said that Renyi's entropy measures a diversity (or dissimilarity) 
within a given distribution ^j- I n Section T2.5I we will see that in parametric 
statistics Fisher information plays a similar role. It will be shown that the latter 
measures a diversity between two statistical populations. 

To find the most fundamental (and possibly irreducible) set of properties 
characterizing Renyi's information it is desirable to axiomatize it. Various ax- 
iomatizations can be proposed |9llfi) . For our purpose the most convenient set 
of axioms is the following JHj : 



1. For a given integer n and given V = {pi 7 p 2 , ■ ■ ■ ,p n } {Pk > 0,Y,kPk = 1): 
1{V) is a continuous with respect to all its arguments. 

2. For a given integer n, T(j>i,p2, ■ ■ ■ ,p n ) takes its largest value for pk = 1/n 
(k = 1, 2, . . . , n) with the normalization X ( i = 1. 

3. For a given q e IR; 1(A nB) = 1(A) + I(B\A) with 

I(B\A) = g" 1 (£ fe Qk {q)g{l{B\A = A k ))), 

and Qk{q) = Pk/ 12kPk (distribution V corresponds to the experiment A). 

4. g is invertible and positive in [0, oo). 

5. T(jpi,p2, . . . ,p n , 0) = I(pi,p2, ■ ■ ■ ,Pn), i-e., adding an event of probability 
zero (impossible event) we do not gain any new information. 

Note particularly the appearance of distribution g(q) in axiom 3. This, so 
called, zooming (or escort) distribution will prove crucial is Sections 12.41 and 151 
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Further characteristics of expressions J5J) were studied extensively in |9llbJ . 
We list here a few of the key ones. 

(a) RE is symmetric: I q (pi, ...,p n )= l g (pk(i),- ■ ■ ,Pk{n)) ; 

(b) RE is nonnegative: X q (V) > ; 

(c) RE is decisive: X q (0, 1) = 2,(1, 0) ; 

(d) For q < 1 Renyi's entropy is concave. For q > 1 Renyi's entropy in not 
pure convex nor pure concave ; 

(e) RE is bounded, continuous and monotonous in q ; 

(f) RE is analytic in q £ (Ciuin =^ f° r 9 = 1 it equals to Shannon's entropy, 
i.e., lim 9 ^i X q = X . 

Despite its formal origin Renyi's entropy proved important in variety of prac- 
tical applications. Coding theory JUj, statistical inference \T7\, quantum me- 
chanics chaotic dynamical systems 19 20 21 22 and multifractals provide 
examples. The rest of Section 2 will be dedicated to applications in multifractal 
systems. For this purpose it is important to introduce the concept of renormal- 
izcd information. 



2.1 Continuous probability distributions - renormalization 

Let us assume that the outcome space (or sample space) is a continuous d- 
dimensional manifold. It is then heuristically clear that as we refine the measure- 
ment the information obtained tends to infinity. Yet, under certain circumstances 
a finite information can be extracted from the continuous measurement. 

To show this we pave the outcome space 1 with boxes of the size I = 1/n. This 
divides the d-dimensional sample space into cells labelled by an index k which 
runs from 1 up to n d . If ^"(x) is a continuous probability density function (PDF), 
the corresponding integrated probability distribution V n = {Pnk} is generated 
via prescription 

Pnk = I ^(x)d d X. (6) 

J k-th box 

Generic form of I q (P n ) it then represented as 

1- q ^Pn) = divergent in n + finite + o(l) , (7) 

where the symbol o(l) means that the residual error tends to for n — > oo. 
The finite part (= X q (JF)) is fixed by the requirement (or by renormalization 
prescription) that it should fulfill the postulate of additivity in order to be iden- 
tifiable with an information measure. Incidentally, the latter uniquely fixes the 
divergent part ^jl as dlog 2 n. So we may write 



I q {Pnk) ~ d\og 2 n + h + o(l) , 
1 For simplicity's sake we consider that the outcome space has volume V = 



1. 



(8) 
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which implies that 

T q {T) =h = Urn (X q (V nk ) - d\og 2 n) = log 2 (J JF«(x)d d x) . (9) 

The latter might be generalized to piecewise-continuous .F(x)'s (Stiltjes integra- 
tion) and to Lebesgue measurable sets Needless to say that Renyi's entropy 
2g(JF) exists iff. the integral on the RHS of © exists. 
Note that © can be recast into a form 

T q {T) = \im{l q {V n )-l q {£ n )). (10) 

n— >oo 

with £ n = {^j,...,^} being the uniform distribution. Expression (|10[) repre- 
sents nothing but Renyi's generalization of the Szilard-Brillouin negentropy. 



2.2 Fractals, multifractals and generalized dimension 

Aforementioned renormalization issue naturally extends beyond simple metric 
outcome spaces (like TR d ). Our aim in this Subsection and the Subsection to 
follow is to discuss the renormalization of information in cases when the out- 
come space is fractal or when the statistical system in question is multifractal. 
Conclusions of such a reneormalization will be applied is Subsection l2.4l 

Fractals are sets with a generally non-integer dimension exhibiting property 
of self-similarity. The key characteristic of fractals is fractal dimension which is 
defined as follows: Consider a set M embedded in a d-dimensional space. Let 
us cover the set with a mesh of ci-dimensional cubes of size l d and let Ni(M) is 
a number of the cubes needed for the covering. The fractal dimension of M is 
then defined as |23I24| 

D = _ nm l^M) (n) 
i^a Inl 

In most cases of interest the fractal dimension coincides with the Hausdorff- 
Besicovich dimension used by Mandelbrot |23| . 

Multifractals, on the other hand, are related to the study of a distribution 
of physical or other quantities on a generic support (be it or not fractal) and 
thus provide a move from the geometry of sets as such to geometric properties of 
distributions. Let a support is covered by a probability of some phenomenon. If 
we pave the support with a grid of spacing I and denote the integrated probability 
in the ith box as Pi, then the scaling exponent a.i is defined |23I24) 

Pi(l) ~ ^ ■ (12) 

The exponent on is called singularity or Lipshitz-Holder exponent. 

Counting boxes N(a) where Pi has oil £ (a, a + da), the singularity spectrum 
/(a) is defined as |23l2l| 

N(a) - r f{a) . (13) 
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Thus a multifractal is the ensemble of intertwined (uni)fractals each with its 
own fractal dimension For further investigation it is convenient to define 

a "partition function" |2.'{| 

Z(q) = $>? = / da'p{a')l-^ a 'n^' . (14) 

i 

In the small I limit the method of steepest descent yields the scaling [2H] 

Z(q)~l T , (15) 

with 

r(q) =min(qa- f(a)), f'(a(q))=q. (16) 

a 

This is precisely Legendre transform relation. So pairs f(a),a and r(q),q, are 
conjugates with the same mathematical content. 

Connection of Renyi entropies with multifractals is frequently introduced via 
generalized dimensions 

These have direct applications in chaotic attractors 19 2UI21I22) and they also 
characterize, for instance, intermittency of turbulence |17I25| or diffusion-limited 
aggregates (DLA) like patterns |2H]. In chaotic dynamical systems all D q are 
necessary to describe uniquely e.g., strange attractors [22] ■ While the proof in [22] 
is based on a rather complicated self-similarity argumentation, by employing the 
information theory one can show that the assumption of a self-similarity is not 
really fundamental JS]- For instance, when the outcome space is discrete then 
all D q with q e [1, oc) are needed to reconstruct the underlying distribution, and 
when the outcome space is d-dimensional subset of IR d then all D q , q G (0, oo), 
are required to pinpoint uniquely the underlying PDF. The latter examples are 
nothing but the information theoretic variants of Hausforff's moment problem 
of mathematical statistics. 



2.3 Fractals, multifractals and renormalization issue 

In a close analogy with Section ini it can be shown ^] that f° r a fractal outcome 
space the following asymptotic expansion of Renyi's entropy holds 

Tq(pkn) w D\og 2 n + h + o{l), (18) 

where D corresponds to the Hausdorff dimension. The finite part h is, as be- 
fore, chosen by the renormalization prescription - additivity of information for 
independent experiments. Then 

J 9 (JF) =h= lim (l q (V n ) - D log 2 n) = lim (l q (V n ) - l q {£ n )) 

n— >oo n— »oo 

-oV^/m*^)' (19) 
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Measure /i in (|19fl is the Hausdorff measure 

i j 7\ id i->o fO if d < D 



Technical issues connected with integration on fractal supports can be found, 
for instance, in |27I28| . Again, renormalized entropy is defined as long as the 
integral on the RHS of l|19|l exists. 

We may proceed analogously with multifractals. The corresponding asymp- 
totic expansion now reads 



l q (p„ k )*-0}-rlog 2 n + h + o(l). (21) 



This implies that 



r(g) 



h = I q (fi v ) = lim X q {Vn) - log 2 n = lim (X,(7>„) - T q {£ n )) 



1 / / -.(g), 



log 2 / d^>(a) . (22) 



(1-9) 

Here the multifractal measure is defined as (51] 

#M- Z $ * {Ititlft. < 23 » 

k-th box k v ' 

It should be stressed that integration on multifractals is rather delicate technical 
issue which is not yet well developed in the literature |28j. 

2.4 Canonical formalism on multifractals 

We shall now present an important connection of Renyi's entropy with mul- 
tifractal systems. The connection will be constructed in a close analogy with 
canonical formalism of statistical mechanics. As this approach is thoroughly dis- 
cussed in we will, for shortness's sake, mention only the salient points here. 

Let us first consider a multifractal with a density distribution p(x). If we use, 
as previously, the covering grid of spacing I then the coarse-grained Shannon's 
entropy of such a process will be 

^n(0) = -E^(0log 2 Pfe(0- (24) 
Important observation of the multifractal theory is that when q = 1 then 

q(l) = g£U /K1 )) = lim l0 f »® = - lim ^# , (25) 

w dq JV v " i^o \og 2 l i^o \og 2 l K 1 

describes the Hausdorff dimension of the set on which the probability is concen- 
trated - measure theoretic support. In fact, the relative probability of the com- 
plement set approaches zero when / — > 0. This statement is known as Billingslcy 
theorem [25] or curdling |23|. 
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For the following considerations it is useful to introduce a one-parametric 
family of normalized measures g(q) (zooming or escort distributions) 

( a D- fo(Q]' - . if(*J (26) 



Because 



,r/ \ [ < da if q < 1 , 

d ^ = {>da-/ q >l] W 
we obtain after integrating (|27[) from a(q — 1) to a(q) that 

So for q > 1 g(g) puts emphasis on the more singular regions of V n , while for 
q < 1 the accentuation is on the less singular regions. Parameter q thus provides a 
"zoom in" mechanism to probe various regions of a different singularity exponent. 

As the distribution 1)26(1 alters the scaling of original V n , also the measure 
theoretic support changes. The fractal dimension of the new measure theoretic 
support of g(q) is 

d h (M^) = Jim V g k (q, I) log 2 Qk(q, I) ■ (29) 

Note that the curdling (|29[1 mimics the situation occurring in equilibrium statis- 
tical physics. There, in the canonical formalism one works with (usually infinite) 
ensemble of identical systems with all possible energy configurations. But only 
the configurations with Ei w {E(T)} dominate at n — > oo. Choice of temperature 
then prescribes the contributing energy configurations. In fact, we may define 
the "microcanonical" partition function as 



dN( ai ) . (30) 



Then the microcanonical (Boltzmann) entropy is 

H(£ (oi) ) = log 2 dN (a, ) = log 2 Z mic , (31) 

and hence 




log 2 £ 



-{f(a))mic- (32) 



Interpreting Ei — —a, log 2 s as "energy" we may define the "inverse temperature" 
1/T = /?/ln2 (note that here k B = l/ln2) as 



7 as 



E=e- In £ Z m i c da 



--}{a i ) = q. (33) 
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On the other hand, with the "canonical" partition function 
Z can = I>(e) 9 = X>-^* , 

i i 

and (3 = gin 2 and Ei = — \og 2 (pi(s)) the corresponding means read 



i(q) = (a) can = ^2 ■ 



-0Ei 



Ezg»(g> £ ) lQ g2K( £ ) 

log 2 e 



^ 2 C „„ l0g 2 £ 



(34) 

(35) 
(36) 



Let us note particularly that the fractal dimension of the measure theoretic 
support dhiM^) is simply f(q). By gathering the results together we have 



micro canonical ensemble 
- unifractals 


canonical ensemble 
- multifractals 




Z C an'; S can = l°g 2 %can ~ q( a ) can log 2 £ 


\Q>/mic &k / Z m i c 


can — ^^fc ^ h /^can 


(f(a))mic = -Smic/ log 2 e 


(f(a))can = -S can /\og 2 e 


g = dS rn i C /dE\ E=Ei 


q = dS can /d(E) can 


(3 = \n2/T = q 


P = bx2/T = q 


Ei = - log 2 Pi = -m log 2 £ 


(E) C an = -(a) can log 2 6 


(f(a))mic = q{a) m ic - t 


(f(a))can = q(a)can ~ T 



Looking at fluctuations of a in the "canonical" ensemble we can establish an 
equivalence between unifractals and multifractls. Recalling Ea. p5|l and realizing 
that 



9 2 (l0g 2 Z C an)/dq 2 = (E 2 )can - {E)l an « (bg 2 £ f , 

<9 2 (r log 2 e)/dq 2 = (da/dq) log 2 e w log 2 e , 
we obtain for the relative standard deviation of "energy" 

1 



^{E*)can-{E)\ 

log 2 e 



y/ {a 2 )can - {a} 2 



0. 



(37) 
(38) 



(39) 



So for small e (i.e., exact multifractal) the a-fluctuations become negligible and 
almost all at equal to (a) ca „. If q is a solution of the equation ai — r'(q) then 
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in the "thermodynamic" limit (e — > 0) a,; ~ (a) can and the microcanonical and 
canonical entropies coincide. Hence 



SVmc ~ - ^2 Qk{q, e) log 2 Qk(q, e) = H(V n 



1/(9) 



The subscript f(q) emphasizes that the Shannon entropy TC(V n ) is basically 
the entropy of an unifractal specified by the fractal dimension f(q). Legendre 
transform then implies that 

nVn)\ nq) « -qa(q) log 2 (e) + (1 - q)X q {V) . (40) 

Employing the rcnormalization prescriptions l|19|) and \\1'2\\ we finally receive that 

^-wl/w ( 41 ) 

So by changing the q parameter Renyie's entropy "skims over" all renormalizcd 
unifractal Shannon's entropies. Renyi's entropy thus provides a unified informa- 
tion measure which keeps track of all respective unifractal Shannon entropies. 

The passage from multifractals to single-dimensional statistical systems is 
done by assuming that the a-interval gets infinitesimally narrow and that PDF 
is smooth. In such a case both a and f(a) collapse to a = /(a) = D and 
q = f'(a) = 1. For instance, for a statistical system with a smooth measure and 
the support space IR <i Eq. 141fl constitutes a trivial identity. We believe that this 
is the primary reason why Shannon's entropy plays such a predominant role in 
physics of single-dimensional sets. Discussion of 141(1 can be found in |lfi| . 



2.5 Renyi's entropy and Fisher's information 

Let us present here an interesting connection which exists between Riemaniann 
geometry on statistical parameter spaces and Renyi entropies. 

Consider a family of PDF's characterized by a vector parameter 9 

T e = {p(x,6);x eM;9e M, a manifold inffi,™} . (42) 

We further assume that p(x, 9) € C 2 . The Gibbs PDF's (with 9i being the inverse 
temperature (3, the external field H, etc.) represent example of (l4*2l . 

To construct a metric on Ai which reflects the statistical properties of the 
family (|4^|l Rao a co-workers [^D] proposed to adopt various measures of dis- 
similarity between two probability densities, and then use them to derive the 
metric. Important class of dissimilarity measures are measures based on infor- 
mation theory. Typically it is utilized the gain of information when a density 
p(x, <fi) is replaced with a density p(x, 9). In the case of Renyi's entropy this is (Hj 

X 9 (0||0) = ^-log 2 / dx f*'^ . (43) 
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The information metric on M. is then defined via the leading order of dissimilarity 
between p(x, 9) and p(x, (9 + d9)), namely 



l q (9\\9 + d9) = ^J2s^) d9 t d9 3 



(44) 



Note that because I q 
we have 



is minimal at 6 = <fi, linear term in l|44|l vanishes. So 



m = 



-urn 



2 In 2 



f dr n(r m gln ^^)glnp(x,g) 



2 In 2 



Fij(p(x,0). 



(45) 



Here is the Fisher information matrix (or Fisher-Rao metric) jolloj . Fisher 
matrix is the only Riemaniann metric which is invariant under transformation 
of variables as well as reparametrization 15 . In addition, the diagonal elements 
of Fisher's information matrix represent the amount of information on 9i in an 
element of a sample chosen from a population of density functions p(x, 9). Due 
to its relation with Cramer-Rao inequality Fisher information matrix plays a 
crucial role in parametric estimation [5]. Let us stress that the latter is used in 
quantum mechanics to formulate information uncertainty relations |5I18| . 



3 Tsallis' entropy 



Tsallis' entropy (or non-extensive entropy, or g-order entropy of Havrda and 
Charvat |15l31p has been recently much studied in connection with long-range 
correlated systems and with non-equilibrium phenomena. Although firstly in- 
troduced by Havrda and Charvat in the cybernetics theory context it was 
Tsallis and co-workers [321 who exploited its non-extensive features and placed 
it in a physical setting. Applications of Tsallis' entropy are ranging from 3- 
dimensional fully developed hydrodynamic turbulence, 2-dimcnsional turbulence 
in pure electron plasma, Hamiltonian systems with long-range interactions to 
granular systems and systems with strange non-chaotic attractors. The explicit 
form of Tsallis' entropy reads 



S a 



(1-9) 



q>0. 



(46) 



This form indicates that S q is a positive and concave function in V. In the 
limiting case q — > 1 one has lim 9 ^i S q = lim^i X q = Ji. In addition, (|46|l obeys 
a peculiar non-extensivity rule 

S q (A n B) = S q {A) + S q {B\A) + (1 - q)S q {A)S q {B\A) . (47) 

It might be proved that the following axioms uniquely specify Tsallis' entropy |33| : 
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1. For a given integer n and given V = {pi 7 p 2 , ■ ■ ■ ,p n } {Pk > 0,Y,kPk = 1)> 
S(V) is a continuous with respect to all its arguments. 

2. For a given integer n, S(j>\,p2, ■ ■ ■ ,p n ) takes its largest value for p k = 1/n 
(fc = l,2,...,n). 

3. For a given q G IR; S(A n B) = S(A) + S{B\A) + (1 - ?)5(A)5(B|i) with 

S(B|A) = £ k e fc (g) S(B\A = A k ), 

and gfc(g) = P^/X^fcPfc (distribution corresponds to the experiment A). 

4. S(p!,p 2 , ■ ■ ■ ,Pn,0) = S(p 1 ,p 2 , . . ■ ,p n )- 

Note that these axioms bear remarkable similarity to the axioms of Renyi's 
entropy. Only the axiom of additivity is altered. We keep here the linear mean but 
generalize the additivity law. In fact, the additivity law in axiom 3 is the Jackson 
sum (or g-additivity) of g-calculus. The Jackson basic number [X] q of quantity 
X is defined as [X] q = (q x — l)/(q— 1). This implies that for two quantities X 
and Y the Jackson basic number [X + Y] q = [X] q + [Y] q + (q - l)[X] q [Y] q , The 
connection with axiom 3 is established when q — > (2 — q). 

The former axiomatics might be viewed as the g-deformed extension of Shan- 
non's information theory. Obviously, in the q — * 1 limit the Jackson sum reduces 
to ordinary <S( Af] B) = S(A)+S(B\A) and above axioms boil down to Shannon- 
Khinchin axioms of classical information theory [33] . 

Emergence of g-deformed structure allows to formalize many calculations. 
For instance, using the g-logarithm 35 , i.e., ln 9 x = (x 1 ^ 9 — 1)/(1 — q), Tsallis' 
entropy immediately equals to the g-deformed Shannon's entropy (again after 
q -»• (2 - q) ), i.e. 

n 

S q (V) = -J2Pk^ qPk . (48) 
fc=i 

The interested reader may find some further applications of g-calculus in non- 
extensive statistics, e.g., in 

Let us finally add a couple of comments. Firstly, it is possible to show 
that Renyi's entropy prescribes in a natural way the renormalization for Tsallis' 
entropy in cases when the PDF is absolutely continuous. This might be achieved 
by analytically continuing the result for renormalized Renyi entropy from the 
complex neighborhood of q = 1 to the entire right half of the complex plane jlri| . 
Thus, if T is the corresponding PDF then 

W) - "a, - - (i^y /„ W - ■ 

Extension to multifratals is more delicate as a possible non-analytic behavior of 
/(a) and r(g) invalidates the former argument of analytic continuation. These 
"phase transitions" are certainly an interesting topic for further investigation. 

Secondly, note that Tsallis and Renyi's entropies are monotonic functions 
of each other and thus both are maximized by the same V . This particularly 
means that whenever one uses MaxEnt approach (e.g., in thermodynamics, image 
processing or pattern recognition) both entropies yield the same results. 
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4 Conclusions 

In this lecture we have reviewed some information-theoretic aspects of general- 
ized statics of Renyi and Tsallis. 

Major part of the lecture - Section 2 - was dedicated to Renyi's entropy. 
We have discussed the information-theoretic foundations of Renyi's information 
measure and its applicability to systems with continuous PDF's. The latter in- 
clude systems with both smooth (usually part of IR d ) and fractal sample spaces. 
Particular attention was also paid to currently much studied multifractal sys- 
tems. We have shown how the Renyi parameter q is related to multifractal sin- 
gularity spectrum and how Renyi's entropy provides a unified framework for all 
unifractal Shannon entropies. 

In cases when the physical system is described by a parametric family of prob- 
ability distributions one can construct a Riemannian metric over this probability 
space - information metric. Such a metric is then a natural measure of diversity 
between two populations. We have shown that when one employs Renyi's infor- 
mation measure then the information metric turns out to be Fisher-Rao metric 
(or Fisher's information matrix). 

In Section 3 we have dealt with Tsallis entropy. Because detailed discussions 
of various aspects of Tsallis statistics are presented in other lectures of this series 
we have confined ourselves to only those characteristics of Tsallis' entropy which 
make it interesting from information-theory point of view. We have shown how 
the g-additive extension of original Shannon-Khinchin postulates of informa- 
tion theory gives rise to g-deformed Shannon's measure of information - Tsallis 
entropy 
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